Neural Networks (multilayer perceptrons) are among the most widely used models in modern machine learning — they power everything from image recognition to language translation. Conceptually, they are the natural stacked generalization of models you already know: if a neuron is "logistic regression in a box," then a neural network is many such boxes stacked into layers so the model can learn rich, internal hierarchical feature representations that linear models cannot express on their own. This unit covers the biological motivation, historical context, architecture, cost function, forward pass (in both scalar and matrix notation), and activation functions that make deep learning possible.
A neural network is a computational model composed of interconnected "neurons" (nodes) organized in layers: input → one or more hidden layers → output layer.
Each hidden-layer node \( j \) receives inputs \( x_1, x_2, \ldots, x_p \) from the previous layer. It first computes a weighted sum plus bias (the pre-activation):
Where \( w_{1,j}, \ldots, w_{p,j} \) are the weights feeding into node \( j \), and \( b_j \) (often written \( \theta_j \)) is the neuron's bias — it controls the baseline activation even when all inputs are zero. Then, crucially, the node applies a non-linear activation function, e.g., the logistic sigmoid:
The output \( a_j \) becomes an input to the next layer, or (if this is the last hidden layer) to the output layer. By stacking layers of non-linear neurons, the network can learn progressively more abstract representations of the input data.
If every neuron used \( f(z) = z \) (identity/"linear activation"), the entire network would collapse mathematically to a single linear transformation. Two weight matrices multiplied together are still one weight matrix, and the model is equivalent to plain linear (or logistic) regression — the depth buys you nothing. The non-linear activation is what makes a deep network strictly more expressive than a shallow one.
| Obs. | Fat score | Salt score | Acceptance |
|---|---|---|---|
| 1 | 0.2 | 0.9 | like |
| 2 | 0.1 | 0.1 | dislike |
| 3 | 0.2 | 0.4 | dislike |
| 4 | 0.2 | 0.5 | dislike |
| 5 | 0.4 | 0.5 | like |
| 6 | 0.3 | 0.8 | like |
We build a small network: input (2 features) → 1 hidden layer (3 units, sigmoid) → output layer (1 unit, sigmoid → probability of "like"). Note the connection to logistic regression: if we removed the hidden layer entirely and connected the inputs directly to the output sigmoid, we would recover plain logistic regression on Fat & Salt. The hidden layer adds representational power — it learns internal features that linear models cannot.
All weights \( w_{i,h_j} \) (input → hidden) and \( w_{h_j,o} \) (hidden → output) plus biases \( b_{h_j} \) and \( b_o \) are initialized to small random numbers near zero. If they were all zero, every neuron would compute the same function and no learning would take place (symmetry problem). Random breaks the symmetry; small ensures activations start in a reasonable range of the sigmoid.
As in logistic regression, we define a loss that measures how far the network's output \( \hat{y} \) is from the true label \( y \). For binary classification it's still Binary Cross-Entropy; what's new is that \( \hat{y} \) is now a composite function of all weights and biases across every layer of the network (and the input features).
Averaged over \( m \) training examples:
where, explicitly, the prediction is the nested composite:
Training is still gradient descent: every parameter (every weight and every bias in every layer) is updated by moving it opposite the gradient of \( J \) with respect to that parameter. In Unit 21 we will compute those gradients via backpropagation using the chain rule. For now, we simply write the update template to see the shape:
Coding the forward pass with for-loops over every weight is slow. Instead, we vectorize using matrix multiplication, which lets GPU hardware (BLAS / cuBLAS) accelerate the computation massively.
| Quantity | Shape | Meaning |
|---|---|---|
| Input vector \( x \) | \( \mathbb{R}^{2 \times 1} \) | Fat, Salt scores |
| Weight matrix \( W_1 \) | \( \mathbb{R}^{2 \times 3} \) | Input → Hidden weights |
| Bias vector \( b_1 \) | \( \mathbb{R}^{3 \times 1} \) | 3 hidden-unit biases |
| Weight matrix \( W_2 \) | \( \mathbb{R}^{3 \times 1} \) | Hidden → Output weights |
| Bias scalar \( b_2 \) | \( \mathbb{R}^{1 \times 1} \) | Single output bias |
With \( m \) examples, you would stack the \( x \) vectors into a \( 2 \times m \) matrix \( X \), and every operation above broadcasts over the batch dimension — no loops required. This vectorization is the reason GPUs make neural networks practical.
Different activations serve different purposes. The right choice depends on: (a) which layer it is (hidden vs. output) and (b) the problem type (regression / binary / multiclass).
| Activation | Formula | Range | Typical Use |
|---|---|---|---|
| Linear / Identity | \( f(z) = z \) | \( (-\infty, +\infty) \) | Output layer only for regression tasks (no squashing needed). Never in hidden layers. |
| Sigmoid / Logistic | \( \sigma(z) = \frac{1}{1+e^{-z}} \) | \( (0, 1) \) | Output layer only for binary classification (probability of the positive class). Historically used in hidden layers but causes vanishing gradients in deep nets. |
| Tanh (Hyperbolic Tangent) | \( \tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}} \) | \( (-1, 1) \) | Hidden layers (better than sigmoid because zero-centered, stronger gradients). Still suffers vanishing gradients at the extremes for very deep networks. |
| ReLU (Rectified Linear Unit) | \( \text{ReLU}(z) = \max(0, z) \) | \( [0, +\infty) \) | Default hidden-layer choice in modern deep learning. Fast, sparse, does not saturate for z > 0. Can "die" (always output 0 for a given neuron — Dying ReLU). |
| Softmax | \( \text{softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}} \) | \( (0, 1) \); sum = 1 | Output layer only for multiclass classification (K > 2 classes). Converts K raw scores into a calibrated probability distribution. |
A beautiful identity you'll use every time you implement backpropagation manually:
Since \( a = \sigma(z) \) is already stored from the forward pass, you get the derivative "for free" — no exp() recomputation needed!
During backpropagation, gradients get multiplied as they flow backward through layers. If these gradients are very small (\(< 1\)), they get smaller and smaller with each layer, eventually becoming nearly zero.
Why it happens: Early layers (close to input) barely learn anything because their gradients are too tiny to cause meaningful weight updates.
Chain rule in backpropagation:
Problem with sigmoid:
Example in a 5-layer network:
Result: Early layers learn very slowly or not at all.
Dying ReLU Problem: If a ReLU neuron's output is always negative (z ≤ 0), its gradient will always be zero, and the neuron will never update its weights. This neuron is effectively "dead."
Solutions:
The exponential function amplifies differences and makes the model more confident in its predictions.
Example: Given output vector: [1, 2, 3]
| Layer | Problem Type | Recommended Activation |
|---|---|---|
| Hidden (any) | Any | ReLU (default) · Tanh (alternative) |
| Output | Regression | Linear (no activation) |
| Output | Binary classification | Sigmoid |
| Output | Multiclass classification | Softmax |
Avoid sigmoid / tanh in the hidden layers of very deep networks — they cause the vanishing gradient problem we'll study in depth in Unit 22.
Tiny network: 5 inputs, 1 hidden layer with 10 units (ReLU), 1 output (sigmoid, binary classification). Count: (a) total number of trainable weights, (b) total number of trainable biases, (c) total parameters.
Layer 1 (Input → Hidden): 5 features × 10 hidden = 50 weights. 10 hidden biases.
Layer 2 (Hidden → Output): 10 hidden × 1 output = 10 weights. 1 output bias.
A. Output layer of a model predicting tomorrow's high temperature (°C). Which activation?
B. Output layer classifying handwritten digits (0–9) into exactly one class. Which activation?
C. Hidden layers in a 20-layer computer-vision model. Which activation is the modern default?
A student argues: "Why not set all weights and biases to 0? That would be simple and we wouldn't have to worry about the random seed." What breaks?
Symmetry-breaking failure. If every neuron in a layer starts with identical weights/biases:
Fix: initialize with small random numbers (Glorot/Xavier for tanh/sigmoid, He for ReLU — covered in Unit 22) so each neuron takes a different learning path from the start.
Input: \( x_1 = 0.6,\ x_2 = 0.4 \). First hidden unit: weights \( w_{1,1} = 0.5,\ w_{2,1} = 0.5 \), bias \( b_1 = -0.5 \). Activation: sigmoid.
Step 1: Pre-activation z₁:
Step 2: Apply sigmoid:
Network architecture: 4 inputs, 1 hidden layer with 8 units (ReLU), output layer with 3 units (softmax, 3-class classification). Mini-batch size of 16 examples stacked as columns: \( X \in \mathbb{R}^{4 \times 16} \).
| Quantity | Shape |
|---|---|
| \( W_1 \) (input → hidden weights) | \( \mathbb{R}^{4 \times 8} \) |
| \( b_1 \) | \( \mathbb{R}^{8 \times 1} \) (broadcasts across 16 examples) |
| \( Z_1 = W_1^T X + b_1 \) | \( \mathbb{R}^{8 \times 16} \) |
| \( A_1 = \text{ReLU}(Z_1) \) | \( \mathbb{R}^{8 \times 16} \) |
| \( W_2 \) (hidden → output weights) | \( \mathbb{R}^{8 \times 3} \) |
| \( b_2 \) | \( \mathbb{R}^{3 \times 1} \) |
| \( Z_2 = W_2^T A_1 + b_2 \) | \( \mathbb{R}^{3 \times 16} \) |
| \( \hat{Y} = \text{softmax}(Z_2) \) | \( \mathbb{R}^{3 \times 16} \) (columns sum to 1) |
True label y = 1 (like). A randomly initialized NN outputs \( \hat{y} = 0.3 \) (predicting dislike with 70% confidence — wrong!).
Before training: y = 1, ŷ = 0.3:
After near-perfect training: y = 1, ŷ = 0.99:
Interpretation: Loss dropped ~120× as the model became confident and correct. The gradient of this loss pushes all weights toward configurations that increase ŷ toward 1 — exactly the behavior we want.
Input x₁ = 1, x₂ = 3. Hidden layer has two neurons with sigmoid activation:
Compute the two activations a₁ and a₂.
Neuron 1: z₁ = 0.3 + 0.2(1) + (−0.1)(3) = 0.3 + 0.2 − 0.3 = 0.2 → a₁ = σ(0.2) ≈ 0.5498.
Neuron 2: z₂ = −1.0 + 0.5(1) + 0.25(3) = −1 + 0.5 + 0.75 = 0.25 → a₂ = σ(0.25) ≈ 0.5622.
You train a 3-hidden-layer network with identity activation f(z)=z everywhere. A friend trains a plain logistic regression on the same data. Both are properly tuned and converged.
3-class output layer. Raw scores before softmax are \( z = [1,\ 2,\ 3] \). Compute \( \text{softmax}(z) \) and verify the outputs sum to 1. Use: \( e^1 \approx 2.718,\ e^2 \approx 7.389,\ e^3 \approx 20.086 \).
Denominator = \( 2.718 + 7.389 + 20.086 \approx 30.193 \).
Check: 0.090 + 0.245 + 0.665 = 1.000 ✓. Notice how softmax amplifies differences — the largest score (3) receives ~2/3 of the probability mass, not 1/2 as in simple linear normalization.
You are building a neural network for each of the following tasks:
Task: What activation function would you use for the output layer in each case?
Given the ReLU activation function \(f(z) = \max(0, z)\), calculate the derivative for the following inputs:
Using the derivative definition:
Calculate the softmax for the following input vector:
z = [1, 2, 3]
Task: Compute softmax(z)
Using the softmax formula:
Step 1: Compute exponentials:
Step 2: Compute softmax for each element:
Verification: 0.090 + 0.245 + 0.665 ≈ 1.000 ✓
Your score: 0 / 5